Self-Decoupling and Ensemble Distillation for Efficient Segmentation

Knowledge distillation (KD) is a promising teacher-student learning paradigm that transfers information from cumbersome teacher to student network. To avoid the training cost of large network, recent studies propose distill knowledge itself, called Self-KD. However, due limitations performance and capacity student, soft-labels or features distilled by barely provide reliable guidance. Moreover, most Self-KD algorithms are specific classification tasks based on soft-labels, not suitable for semantic segmentation. alleviate these contradictions, we revisit label feature problem in segmentation, Self-Decoupling Ensemble Distillation Efficient Segmentation (SDES). Specifically, design decoupled prediction ensemble (DPED) algorithm generates with multiple expert decoders, (DFED) mechanism utilize more important channel-wise maps encoder learning. The extensive experiments three public segmentation datasets demonstrate superiority our approach efficacy each component framework through ablation study.